Fusion of Children's Speech and 2D Gestures when Interacting with 3D Embodied Conversational Characters

Most of the existing multimodal prototypes enabling users to combine 2D gestures and speech are task-oriented. They help adult users to solve particular information tasks often in 2D standard Graphical User Interfaces. This paper describes the NICE HCA system which aims at demonstrating multimodal conversation between humans and embodied historical and literary characters. The target users are 10-18 years old children and teenagers. We discuss issues in 2D gestures recognition and interpretation, temporal and semantic dimensions of input fusion, ranging from systems and component design through technical evaluation and user evaluation with two different groups. We observed that the recognition and the understanding of spoken deictics revealed to be quite robust and spoken deictics were always used in multimodal input. We identified the causes of the most frequent failures of input fusion, i.e., end of speech management in the speech recogniser, gestures on nonreferenceable objects, and input gesturing while the character is preparing to speak. We suggest possible improvements for removing these errors and conclude on the knowledge provided by the NICE HCA system on how children gesture and combine their 2D gestures with speech when conversing with a 3D character in such a multimodal conversation oriented system.